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Sweeping  Series  of  Enhancements 
Engulf  NAVO  MSRC 


The  next  few  months  will  be  notable  ones  here  at 
the  NAVO  MSRC.  We  are  completing  a  sweeping 
series  of  center  enhancements,  designated  as 
Technology  Insertion  for  FY02  (T 1-02),  across 
several  major  technology  areas  within  the  MSRC. 
The  centerpieces  of  T 1-02  are  the  acquisition  of  a 
new  IBM  POWER4  HPC  system  and  completion 
of  the  NAVO  MSRC  Remote  Storage  Facility 
(RSF).  When  complete,  the  T 1-02  enhancements 
will  provide  almost  8.5  teraflops  of  aggregate 
peak  computing  capability  with  commensurately 
balanced  storage  and  networking  capabilities. 

The  RSF  will  permit  us  to  offer  what  is  perhaps 
one  of  the  most  resilient  and  best  performing 
HPC  storage  environments  in  the  world  today. 
This  enormous  computational  capability,  coupled 
with  a  sustained  11-year  NAVOCEANO  focus  on 
supporting  the  largest  and  most  demanding  DoD 
computational  applications,  will  continue  to 
enable  unparalleled  advances  in  the  DoD  science 
and  technology  areas  served  by  the  H  PC 
Modernization  Program  (HPCMP). 

With  all  of  this  diverse  computational  capability 
that's  been  fielded  across  20+  shared  resource 
centers  (SRCs)  by  the  HPCMP,  it  has  become  crit¬ 


ically  important  for  us  to  redouble  our  efforts  in 
assessing  and  implementing  common  user  envi¬ 
ronments,  practices,  and  tools  within  and  across 
the  SRCs.  Your  continuing  individual  and  collec¬ 
tive  user  feedback  makes  it  clear  that  you  consid¬ 
er  this  to  be  one  of  your  highest  priorities  for  the 
SRCs.  In  response,  the  SRCs  have  intensified  and 
made  significant  progress  toward  strategic  cross¬ 
cutting  collaborative  efforts  in  technical  areas 
such  as  mass  storage  and  archival,  metacomput¬ 
ing,  HPCMP-wide  shared  information  environ¬ 
ments,  and  security.  H  ere  at  the  NAVO  MSRC, 
we  are  redoubling  our  efforts  to  supplement 
those  activities  by  strengthening  the  links  to  the 
new  HPCMP  Programming  Environmentand 
Training  (PET)  program  that's  more  tightly 
focused  than  ever  on  user  environment,  tools, 
and  productivity. 

My  staff  and  I  look  forward  to  seeing  you  in  J  une 
at  the  2002  HPCMP  Users'  Conference  in  Austin, 
Texas.  As  always,  please  take  every  opportunity 
to  let  us  know  how  we  can  better  serve  you  — 
your  feedback  is  critically  important  to  us  and  to 
the  HPCMP. 


About  the  Cover: 


This  image  was  generated  by  the  POP  model  running  on  HABU,  an  IBM  RS/6000  SP  located  atthe  NAVO  MSRC.  The 
simulation  represents  a  state-of-the-art  eddy-resolving  effort  that  will  eventually  be  used  as  the  ocean  component  of  a 
coupled  global  air/ocean/ice  prediction  system  for  N  avy  needs  as  well  as  for  short-term  climate  studies. 
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The  Naval  Oceanographic  Office  (NAVO) 
Major  Shared  Resource  Center  (MSRC): 
Delivering  Science  to  the  Warfighter 

The  NAVO  MSRC  provides  Department  of 
Defense  (DoD)  scientists  and  engineers  with  high 
performance  computing  (HPC)  resources,  includ¬ 
ing  leading  edge  computational  systems,  large- 
scale  data  storage  and  archiving,  scientific  visuali¬ 
zation  resources  and  training,  and  expertise  in  spe¬ 
cific  computational  technology  areas  (CTAs). 
These  CTAs  include  Computational  Fluid 
Dynamics  (CFD),  Climate/Weather/Ocean 
Modeling  and  Simulation  (CWO),  Environmental 
Quality  Modeling  and  Simulation  (EQM), 
Computational  Electromagnetics  and  Acoustics 
(CEA),  and  Signal/Image  Processing  (SIP). 
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Active  Swirl  and  Fuel  Modulation  to 
Control  Combustion  Instability  in  Gas 
Turbine  Engines 

Chris  Stone  and  Suresh  Menon,  Georgia  Institute  of  Technology 


Combustion-generated  pollutants  such 
as  nitrous  oxide  (N20),  carbon  monox¬ 
ide  (CO),  unburned  hydrocarbons 
(UHC),  and  soot  can  be  reduced  signif¬ 
icantly  if  lean  burning  combustion  sys¬ 
tems  can  be  used  in  gas  turbine 
engines.  However,  as  the  fuel-air  mix¬ 
ture  becomes  lean,  small  perturbations 
in  the  flow  can  change  the  unsteady 
heat  release  pattern,  and  if  the  heat 
release  is  in-phase  with  the  acoustic 
fluctuations  in  the  combustor,  the  pres¬ 
sure  (acoustic)  fluctuations  can  grow 
into  high-amplitude,  low-freguency 
oscillations.  These  oscillations  can 
cause  flame  extinction  (often  called 
Lean  Blowout,  or  LBO),  reduce  engine 
life  span,  and/or  cause  catastrophic 
structural  damage  under  extreme  cir¬ 
cumstances.  If  these  oscillations  can  be 
controlled  and  even  leaner  mixtures 
can  be  burned  stably,  then  not  only  can 
emissions  be  reduced  but  the  over  fuel 
consumption  can  also  be  decreased  sig¬ 
nificantly.  Since  these  oscillations  occur 
in  a  dynamic  manner,  passive 


control  technigues 
(e.g.,  using  struc¬ 
tural  changes) 
cannot  deal  with 
all  possible 
changes,  and 
therefore,  in  order 
to  suppress  these 
oscillations,  active 
control  strategies 
are  needed.  This 
article  describes  the 
study  of  two  active  con¬ 
trol  methods  applied  to  pre¬ 
mixed  combustion  in  a  gas  tur¬ 
bine  engine:  modulation  of  the 
swirl  imposed  on  the  incoming  fuel- 
air  mixture  (which  is  at  a  fixed  equiva¬ 
lence  ratio)  and  modulation  of  the  fuel 
content  in  the  premixed  mixture  for  a 
fixed  swirl  condition.  These  studies 
have  been  conducted  using  a  state-of- 
the-art  Large-Eddy  Simulations  (LES) 
model  developed  at  the  Georgia 
Institute  of  Technology. 

The  first  active  control  techmgue  stud¬ 
ied  here  employs  modulation  of  the 
inlet  swirl  velocity  of  the  incoming 
premixed  mixture.  In  a  typical 
gas  turbine  engine,  swirl  is 
introduced  into  the  air 
from  the  compressor 
by  swirl  vanes  (typical¬ 
ly  small  airfoils  at  high 
angles  of  attack)  in  the 
premixer  located 


Figure  1(b) 

upstream  of  the  combustor.  Fuel  is  then 
injected  into  this  swirling  airflow  in  the 
premixer  so  that  a  swirling,  premixed 
fuel-air  mixture  enters  the  combustor. 
Swirling  inflow  is  used  in  dump  com¬ 
bustors  to  exploit  a  fluid  dynamic  phe¬ 
nomenon  called  Vortex  Breakdown 
(VB )  that  makes  the  flame  zone  com¬ 
pact  and  stabilizes  the  flame.  As  the 
swirling  flow  expands  into  the  combus¬ 
tor,  an  adverse  pressure  gradient  is 
formed  in  the  flow,  and  this  slows  down 
the  axial  motion  of  the  mixture.  For  a 
given  geometry,  when  the  swirl  intensi¬ 
ty  exceeds  a  critical  value,  a  recircula¬ 
tion  bubble  is  formed  (called  VB)  along 
the  centerline  of  the  combustor.  This 
bubble  acts  as  an  effective  bluff  body  in 


Figure  1(a) 


Figures  1(a)/(b).  Swirl  modulation  effect  on  flame-vortex 
dynamics:  (a)  high-swirl  condition,  (b)  low-swirl  condition. 
At  the  high-swirl  state,  the  vortex  rings  (purple)  rapidly 
break  down,  and  the  flame  surface  (gray)  is  compact.  On 
the  other  hand,  at  low  swirl,  the  vortex  rings  are  more 
coherent,  and  they  drag  the  flame  with  them,  making  the 
flame  longer.  No  VB  occurs  when  the  swirl  is  low,  and 
therefore,  the  flame  in  the  low-swirl  case  oscillates  and  is 
unstable,  whereas  at  high  swirl,  VB  stabilizes  the  flame. 
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the  flow,  and  along  the  centerline,  the 
flow  from  the  inlet  actually  stops  (stag¬ 
nates)  just  upstream  of  the  bubble.  This 
phenomenon  helps  to  stabilize  the 
flame  just  upstream  of  the  stagnation 
point.  However,  in  current  gas  turbine 
engines,  since  the  swirl  vanes  are  fixed, 
the  swirl  introduced  into  the  flow  is  stat¬ 
ic  and  is  typically  optimized  for  only 
one  operating  condition.  This  study 
explores  how  swirl  magnitude  can  be 
actively  changed  to  stabilize  the  com¬ 
bustion  as  the  fuel  air  mixture  is 
changed  from  rich  to  lean. 

The  second  techmgue  studied  here  is 
the  dynamic  adjustment  of  the  fuel 
content  in  the  fuel-air  mixture.  For  a 
fixed  mass  flow  rate  and  a  fixed  inlet 
swirl  intensity,  we  dynamically  change 
the  fuel-air  mixture  ratio  by  changing 
the  eguivalence  ratio  of  the  incoming 
mixture.  The  goal  of  this  study  is  to 
determine  how  changing  the  fuel  con¬ 
tent  (which,  in  turn,  will  affect  the 
amount  of  heat  released  and  peak  tem¬ 
perature  reached)  will  affect  the  com¬ 
bustion  dynamics  in  the  combustor. 

The  LES  methodology  is  used  here  to 
simulate  the  combustor  response  to 
both  control  techmgues.  In  LES,  the 
large  scales  of  motion  are  fully  resolved 
in  both  space  and  time,  and  only  the 
small  scales  (smaller  than  the  grid  reso¬ 
lution)  are  modeled.  Premixed  com¬ 
bustion  is  modeled  using  a  thin-flame 
model  in  which  a  scalar  field  that  repre¬ 
sents  the  propagating  flame  front  is 
evolved,  along  with  the  LES  momen¬ 
tum  and  energy  transport.  To  model 
the  fuel-air  mixture  change  (in  the  sec¬ 
ond  control  techmgue),  an  additional 
(conserved)  scalar  eguation  is  mod¬ 
eled,  which  tracks  the  change  in  local 
eguivalence  ratio.  The  local  eguiva¬ 
lence  ratio  is  then  used  to  calculate  the 
burning  rate  and  combustion  tempera¬ 
ture. 

The  LES  algorithm  (LESLIE3D)  is  par¬ 
allel  (using  Message  Passing  Interface) 
and  performs  very  well  on  a  variety  of 
distributed  (CrayT3E  (SEYMOUR), 
RS/6000  SP3  (H ABU),  etc.)  and 


shared-memory  parallel  platforms  (SGI 
Origin  2000/3800).  In  fact,  due  to  its 
scalability,  LESLIE3D  is  now  a  bench¬ 
mark  code  for  all  Department  of 
Defense  (DoD)  High  Performance 
Computing  (H  PC)  sites  and  is  being 
used  to  evaluate  new  systems'  per¬ 
formances.  The  current  study 
employed  1.3  million  grid  cells  and 
reguired  approximately  1000  CPU 
hours  (on  HABU)  for  one  flow-through 
time.  Typically,  10-15  flow-through 
times  are  simulated  to  obtain  sufficient 
data  for  statistical  analysis. 

The  first  investigation  looked  at  chang¬ 
ing  the  level  of  swirl  imposed  at  the 
inlet.  In  an  open-loop  study  conducted 
earlier,  the  pressure  fluctuation  was 
shown  to  reduce  when  swirl  intensity 


was  increased.  To  dynamically  repro¬ 
duce  this  effect,  a  simulation  was  start¬ 
ed  with  a  low-swirl  S  (0.56),  and  then 
swirl  was  steadily  increased  so  that  S 
reached  a  final  value  of  1.12.  Analysis 
shows  that  after  a  small  delay,  the  pres¬ 
sure  oscillation  amplitude  dropped  by 
6.6  decibels  (dB).  Figures  1  and  2, 
respectively,  show  the  effect  of  swirl  on 
vortex-flame  dynamics  and  on  the 
mass  flow-rate  fluctuations.  Results 
show  that  for  a  fixed  fuel-air  mixture, 


an  increase  in  swirl  stabilizes  the  flame 
and  reduces  the  fluctuation  in  both  the 
inflow  mass  flow  rate  and  in  the  pres¬ 
sure  fluctuations. 

The  governing  physics  of  combustion 
instabilities  is  guite  complicated  owing 
to  the  fact  that  not  one,  but  several 
physical  processes  are  coupled  and  in 
contention.  The  dominant  process  for 
combustion  instabilities  is  the  interac¬ 
tion  of  acoustic  waves  (pressure  fluctua¬ 
tions)  with  the  unsteady  heat  source  (in 
this  case,  the  flame  surface).  When  the 
pressure  (p1)  and  heat-release  fluctua¬ 
tions  (q1)  are  "in-phase"  (the  product  of 
the  two  being  positive),  energy  is 
added  to  the  instabilities.  As  the  fuel 
concentration  is  decreased,  two  impor¬ 
tant  controlling  parameters,  the  com¬ 


bustion  temperature  (flame  tempera¬ 
ture)  and  the  burning  velocity  (flame 
speed),  are  subseguently  reduced.  The 
nearer  one  is  to  the  LBO  limit,  the 
more  pronounced  is  this  effect  (See 
Figure  3a).  At  a  lower  burning  velocity, 
unsteady  flames  are  more  susceptible 
to  large-scale  flow  perturbations  since 
the  recovery  time  (i.e.,  the  time  need¬ 
ed  to  return  to  the  mean  or  stable 

Article  Continues... 


Figure  2.  Mass  flux  measured  at  the  inlet  during  swirl  modulation.  At  low  swirl, 
the  fluctuating  mass  flux  is  nearly  100%  but  drops  sharply  as  the  swirl  vane 
angle  is  increased  (i.e.,  when  the  swirl  velocity  is  increased).  The  pressure  fluc¬ 
tuation  in  the  combustor  is  also  decreased  as  the  swirl  intensity  is  increased. 
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Figure  3(a) 


Figure  3(a)/(b).  Instantaneous  view 
of  fuel  modulation  control.  Shown 
are  the  flame  surface  (red  isosur¬ 
face),  vortex  rings  (gray  rings),  prod¬ 
uct  temperature  (color  contours), 
and  fluctuating  pressure  signal 
(shown  below  the  figure).  In  (a),  the 
fuel  mixture  entering  the  combustor 
has  become  lean,  and  this  results  in 
a  decrease  in  temperature  (blue 
core  region),  and  the  pressure  oscil¬ 
lation  increases.  This  time  corre¬ 
sponds  to  the  far  right  of  the  p'(t) 
signal.  In  (b),  the  inflow  fuel  mixture 
equivalence  ratio  is  increased  (rich 
mixture),  and  as  a  result,  the  flame 
temperature  increases,  resulting  in  a 
hotter  core  region  (red  temperature 
contours).  At  this  phase  of  the  con¬ 
trol  cycle,  the  pressure  fluctuations 
begin  to  decrease  (first  quarter  of 
the  p'(t)  signal). 


location)  is  lengthened.  In  the  pres¬ 
ent  simulation,  the  magnitude  of 
flame  fluctuations  is  reduced  at  high¬ 
er  burning  velocities  (See  Figure  3b). 
Another  contributing  factor  to  the 
increased  stability  is  the  modification 
of  the  relevant  time  scales.  As  the 
combustion  temperature  is  altered, 
the  acoustic  velocity  (speed  of 
sound)  is  also  modified,  resulting  in  a 
shift  in  the  phase  between  the  pres¬ 
sure  and  heat-release  fluctuations. 
This  effect  can  be  significant  in  either 
increasing  or  decreasing  the  instabili¬ 
ties,  depending  on  the  phase 
between  the  p1  and  q1.  In  some  com¬ 
bustion  devices,  instabilities  are  more 
intense  for  higher  equivalence  ratios 
due  to  this  effect,  while,  as  simulated 
here,  the  reverse  may  also  occur. 

Simulations  such  as  these  can  be 
used  to  understand  the  complex 
nature  of  combustion  instability  that 
is  a  result  of  nonlinear  interactions 
between  the  three  modes  of  wave 
motion  in  a  compressible  reacting 


flow  in  a  confined  domain:  acoustic 
waves,  vortex  motion,  and  unsteady 
heat  release  due  to  combustion. 
Active  control  of  this  instability,  as 
demonstrated  here,  can  then  be  used 


not  only  to  achieve  stable  combus¬ 
tion  in  a  regime  that  is  not  possible 
otherwise,  but  also  to  understand 
how  the  dynamics  have  been 
changed  due  to  control  actuation. 
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NAVO  MSRC  Visualization  Center 
Supports  the  Retrieval  of  the  Ehime  Maru 

Pete  Gruzinskas,  NAVO  MSRC  Visualization  Center 


On  9  February  2001,  a  collision 
between  a  U.S.  Navy  submarine 
and  a  J  apanese  fishing  trawler  sent 
the  fishing  vessel  to  the  bottom  of 
the  Pacific  Ocean  about  10  miles 
south  of  Diamond  H  ead, 

Oahu,  H awaii.  The  decision 
was  made  to  retrieve  the  ves¬ 
sel,  which  lay  in  approxi¬ 
mately  1800  feet  of  water.  A 
Crisis  Action  Team  was 
formed,  and  the  NAVO 
MSRC  Visualization  Center 
was  asked  to  provide  visuali¬ 
zation  support  for  the  Ehime 
Maru  Retrieval  Operation. 

There  are  many  support 
aspects  in  an  operation  like 
this,  but  the  first  challenge 
was  to  visualize  the  high-res¬ 
olution  model  output  gener¬ 
ated  by  the  Shallow-Water 
Analysis  and  Forecast  System 
(SWAFS)  a  three-dimension¬ 
al  (3D)  ocean  circulation 
model  that  produces  time- 
series  images  with  correct 


speed  and  direction,  temperature, 
and  salinity  fields.  Initially,  a  2-km 
grid  that  covered  the  entire 
Hawaiian  Islands  was  generated,  but 
this  model  provided  input  to  a  high¬ 


er  resolution  (500-m)  nest,  that 
bounded  the  operation  area  just 
south  of  Oahu.  The  model  was  run 
on  the  newest  and  most  powerful 
supercomputer  at  the  NAVO  MSRC, 
the  RS/6000  SP3  (HABU). 

Part  of  the  NAVO  MSRC 
Visualization  Center  mission  is 
the  development  of  analytical 
software  environments  for  the 
Department  of  Defense  (DoD) 
research  community.  Some  of 
these  tools,  designed  to  ana¬ 
lyze  ocean  model  output, 
were  modified  to  analyze  the 
Navy's  operational  model  out¬ 
put.  These  software  tools  pro¬ 
vided  significant  diagnostic 
capability,  which  assisted  in 
the  validation  of  the  model 
output  in  a  very  complex 
environment. 

While  virtual  environments 
may  not  provide  all  of  the 
answers  to  data  analysts,  the 

Article  Continues... 


Screen  capture  of  SWAFS  probe  compared  to 
the  ADCP  current  measurements. 


SWAFS  surface  currents  in  the  operational  area. 


Vertical  profile  of  SWAFS  over  the  Ehime  Maru. 
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fact  of  the  matter  is  the  real  world 
environment  is  in  3D.  Features  with¬ 
in  the  environment  have  3D  struc¬ 
ture  that  can  be  difficult,  if  not 
impossible,  to  realize  in  two-dimen¬ 
sional  (2D)  space.  The  same  tech¬ 
nology  that  was  built  to  help 
research  and  development  modelers 
scrutinize  their  model  output 
was  applied  to  an  opera¬ 
tional  scenario  where  the 
speed  and  direction  of 
the  ocean  currents  were 
critical  factors. 

An  additional  feature 
added  to  the  application, 
unique  to  the  Ehime 
Maru  retrieval  operation, 
was  the  display  of  an 
Acoustic  Doppler  Current 
Profiler  (ADCP)  buoy. 

The  display  of  the  ADCP 
data  provided  analysts  a 
direct  comparison 
between  model  output 
and  in-situ  current  meas¬ 
urements  in  near-real 
time.  Another  significant 
feature  of  the  virtual 
environments  built  for 


ocean  model  analysis  is  portability 
—  the  ability  to  run  on  a  variety  of 
hardware  architectures,  including 
laptop  computers. 

Portability  is  accomplished  by  savvy 
application  of  graphics  techniques, 
as  well  as  smart  Input/Output  and 
memory  management.  This  ability 


was  critical  to  the  provision  of  an 
analysis  environment  and  data  to 
forward-deployed  personnel  on¬ 
scene  at  the  recovery  site.  Other 
support  products  were  developed  to 
render  the  recovery  area  in  3D  from 
high-resolution  bathymetry  (water 
depth)  data  to  delineate  critical 

areas.  Conceptual  animations 
were  built  to  demonstrate 
the  mechanics  of  an 
extremely  difficult  recovery 
operation. 

The  operation  was  success¬ 
ful,  and  all  mission  objec¬ 
tives  were  accomplished. 
This  recovery  operation 
demonstrates  the  excellent 
synergy  that  has  developed 
between  the  operational 
Navy  and  the  DoD 
research  infrastructure  built 
by  the  H  igh  Performance 
Computing  Modernization 
Office  (H  PCMO),  and  is  a 
stellar  example  of  the 
NAVO  MSRC  realizing  the 
H  PC M 0  maxim  of 
"Delivering  Science  to  The 
Warfighter." 


3D  shaded  relief  of  OPAREA  using  high-resolution 
bathymetry  collected  by  the  USNS  SUMNER.  Imagery 
shows  where  the  ship  was  resting,  the  shallow-water 
retrieval  area,  and  the  final  relocation  site. 


Screen  capture  showing  ocean  currents  at  depth  Another  snapshot  of  currents  at  depth.  Note  the  leg- 

over  an  exaggerated  (3:1)  model  of  the  Ehime  Maru.  end  has  been  "tuned"  to  accentuate  faster  currents. 
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Still  images  from  conceptual 
animation  showing"  retrieval  of 
the  Ehime  Maru  by  the  vessel 
Rockwater. 


Still  image  from  conceptual  animation  showing  the  lift  pro¬ 
cedure.  A  conceptual  animation  was  generated  depicting 
how  the  towing  harness  would  be  attached  to  the  Ehime 
Maru,  which  rested  in  approximately  1800  feet  of  water. 


Still  image  showing  Ehime  Maru  retrieval  over  real 
bathymetry  data.  The  animation  was  designed  to  concep¬ 
tualize/visualize  what  the  recovery  would  look  like  in  the 
context  of  the  real  data  collected  by  the  USNS  SUMNER. 
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Knowledge  Management:  A  Case  for 
Intelligent  Data  Archives 

Douglas  D.  Cline,  Ph.D.,  Program  Manager,  Northrop  Grumman  Information  Technology 


There  is  little  doubt  that  one  of  the 
most  significant  challenges  facing  the 
Department  of  Defense  (DoD)  H  igh 
Performance  Computing 
M  odernization  Program  today  is  intel¬ 
ligently  dealing  with  the  sheer  magni¬ 
tude  of  data  that  are  created,  collect¬ 
ed,  analyzed,  and  manipulated 


As  the  data  archive  approaches  the 
petabyte  range,  it  is  clear  that  data 
management  initiatives  must  be 
undertaken  by  the  DoD  to  create 
structured,  intelligent  data  archives 
that  provide  a  broad  range  of  func¬ 
tionality.  Such  initiatives  will  prove 
difficult  to  implement  without  first 
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Figure  7.  Conceptual  Framework  for  the  Current  MSRC  Archival  Storage 
System. 


throughout  the  MSRCs  and 
Distributed  Centers  (DCs).  Increasing 
computational  capabilities,  as  demon¬ 
strated  by  faster  H  PC  systems  and 
larger  system  memories,  enable  high- 
fidelity  simulations  that  provide  critical 
information  to  support  the  nation's 
warfighters.  In  addition,  through 
advances  in  sensor  technologies,  sci¬ 
entific  data  are  being  collected  and 
archived  at  an  unprecedented  rate. 
These  two  factors  create  what  is 
known  as  "infrastress"  —  a  fundamen¬ 
tal  stress  applied  to  the  current  systems 
and  data  storage  infrastructure.  As  of  1 
March  2002,  the  NAVO  MSRC  data 
archive  surpassed  375  terabytes  (TB), 
with  a  projected  growth  rate  of 
approximately  14  TB  per  month. 


empowering  end-users  with  a  suite  of 
high-level  data  management  tools  for 
organizing,  browsing,  and  accessing 
long-term  archival  data.  Development 
of  these  tools  will  require  new  per¬ 
spectives  on  the  MSRC  program  to 
leverage  existing  storage  architectures 
with  newer,  more  robust  technologies. 

Traditional  Archival  Storage 

Archival  storage  provides  the  means 
to  preserve  large-scale  computational 
and  experimental  data  indefinitely.  A 
conceptual  framework  for  a  traditional 
archival  storage  system  is  shown  in 
Figure  1,  with  "Data"  representing  the 
foundation  of  the  system.  The  various 
infrastructure  elements  —  tape  silos, 
tape  media,  Hierarchical  Storage 


Management  (HSM)  software,  net¬ 
working,  and  fileservers  —  are  all  crit¬ 
ical  components  of  the  system. 
Scalability  is  essential  if  future  capaci¬ 
ty  demands  are  to  be  met  through 
incremental  expansion  of  the  compo¬ 
nents.  As  data  capacity  expands  with 
time,  so  do  related  storage  costs.  If  the 
growth  in  data  storage  is  exponential, 
then,  arguably,  storage  costs  will  also 
grow  exponentially.  Overtime,  "data 
inertia"  can  become  so  great  that  the 
flexibility  needed  to  migrate  the  active 
archive  to  newer,  lower  cost  storage 
technologies  is  problematic  at  best. 

From  an  end-user's  perspective,  the 
traditional  archival  system  is  inherent¬ 
ly  limited  when  supporting  higher 
level  access  to  data.  Generally,  access 
to  data  stored  in  the  archive  is  low- 
level  via  HSM  -specific  commands.  For 
example,  get /path/file,  put/path/file, 
and  copy  /path/file  are  all  command- 
driven  directives  for  accessing 
archived  data.  Each  directive  requires 
specialized  knowledge  of  the  HSM 
commands  (e.g. ,  get,  put,  copy)  as 
well  as  a  specific  pathname  that 
points  to  a  particular  file  (e.g., 
/path/file).  This  knowledge  is  typically 
limited  to  a  single  user  or  small  com¬ 
munity  where  the  specialized  knowl¬ 
edge  has  been  shared.  An  individual 
outside  this  group,  not  privileged  with 
this  special  knowledge,  would  find  it 
very  difficult  to  locate  the  data  set, 
much  less  extract  useful  information 
from  it.  Therefore,  one  of  the  primary 
limitations  of  the  traditional  storage 
archive  is  its  inability  to  support 
"Information  Discovery,"  a  critical 
missing  capability  for  long-term  per¬ 
sistent  data  storage. 

Article  Continues... 
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Knowledge  Management  Systems 

By  leveraging  the  M  SRC's  current 
storage  infrastructure,  higher  level 
functionality  can  be  achieved  with 
nominal  investment  to  create  a  more 
robust,  intelligent  data  archive.  For  the 
purposes  of  this  article,  the  more  gen¬ 


eral  type  of  archival  storage  will 
be  referred  to  as  a  Knowledge 
M  anagement  System. 

In  principle,  the  Knowledge 
M  anagement  System  should  be  capa¬ 
ble  of  supporting  Information 
Discovery,  including  browsing,  search¬ 
ing,  and  retrieving  archived  data.  The 
conceptual  framework  of  a  Knowledge 
Management  System  is  shown  in 
Figure  2.  All  essential  infrastructure 
elements  of  the  archival  storage  system 
are  present,  as  in  Figure  1.  Flowever, 
a  new  foundation  layer  is  introduced 
in  our  Knowledge  framework, 

M  iddleware,  which  supports  a  set  of 
higher  level  infrastructure  elements. 

The  M  iddleware  provides  a  software 
layer  to  interprethigh-level  data 
access  functions  into  low-level  FH  S  M  - 
specific  commands.  Moreover,  it  acts 
as  an  interface  between  sophisticated 


software  tools  and  applications  to  iso¬ 
late  them  from  the  implementation 
details  of  the  underlying  storage  archi¬ 
tecture. 

M  iddleware  typically  consists  of  Object 
Brokers  such  as  Common  Object 
R eg u est  B  ro  ker  A  rch  i tectu  re 


(CORBA),  or,  more  specific  to  storage 
architectures,  the  Storage  Resource 
Broker  (SRB),  developed  at  the  San 
Diego  Supercomputing  Center.  In  the 
Knowledge  M  anagement  framework, 
changes  to  the  storage  architecture 
can  be  introduced  without  impacting 
the  functionality  of  user  tools  and 
applications,  thereby  protecting  soft¬ 
ware  investments  and  preserving 
operational  continuity. 

Of  critical  importance  to  the 
Knowledge  M  anagement  System  is 
metadata,  or  "Data  about  the  Data." 
The  metadata  represents  a  database 
of  data  attributes,  with  linkages  to 
physical  file  locations  in  the  archive. 
Data  attributes  may  contain  informa¬ 
tion  such  as  data  types,  data  format 
or  representation,  data  descriptors, 
and  date  of  creation.  It  is  through  the 
metadata  that  higher  level  functions 


such  as  Information  Discovery  are 
supported,  by  referencing  data  files 
through  their  data  attributes  rather 
than  physical  file  location. 

Beyond  Data  Archives  - 
Digital  Libraries 

This  article  has  examined  the  basic 
framework  for  two  storage  systems  — 
a  traditional  storage  archive  represent¬ 
ing  the  M SRC's  current  architecture 
and  a  more  robust  framework  repre¬ 
senting  a  Knowledge  Management 
System.  The  Knowledge  Management 
System  supports  a  broad  range  of 
data  access  tools  and  applications  to 
facilitate  data  manipulation  from 
remote  clients.  The  key  aspect  of  the 
Knowledge  M  anagement  System  is 
that  it  leverages  the  M  SRC's  existing 
storage  infrastructure  (i.e.,  silos,  file- 
servers,  and  networks)  by  introducing 
M  iddleware  to  deliver  a  new  class  of 
higher  level  data  services  to  the  end- 
user. 

Beyond  these  simple  frameworks  lie 
more  sophisticated  storage  architec¬ 
tures.  Digital  libraries  offer  the  poten¬ 
tial  to  create  online  knowledge  centers 
for  discipline-specific  application  areas 
that  are  important  to  the  DoD.  These 
libraries  generally  provide  shared 
information  resources,  coupled  with 
high-speed  networks,  to  deliver  high- 
level  information  content  directly  to 
the  desktop.  Client-based  tools  and 
applications  can  be  specifically 
designed  to  enable  data  manipula¬ 
tions  for  analysis  and  interpretation. 

The  technical  challenges  posed  by 
large-scale  data  management  pres¬ 
ents,  in  reality,  new  opportunities  for 
infrastructure  development  within  the 
DoD  modernization  program.  In 
exploiting  these  opportunities,  new 
requirements  and  capabilities  will  be 
defined  that  will  ultimately  deliver  a 
new  class  of  data  services  for  users  of 
the  MSRCs  and  DCs. 


A 


Figure  2.  Conceptual  Framework  for  the  MSRC  Knowledge  Management 
System. 
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Improvements  in  the  Use  of  Color  Wheel 
Visualization  of  Global  Modeling  Data 


Ludwig  Goon,  NAVO  MSRC  Visualization  Center 


Figure  1 .  Global  image  of  POP  model  data  diplayed  using  the  original 

color  wheel  method. 


The  color  wheel  method  of  two-dimen¬ 
sional  (2D)  vector  field  visualization 
uses  the  H  ue-Saturation-Value  (H  S V) 
color  space  to  map  vector  direction  to 
hue  (color)  and  magnitude  to  saturation 
and  value.  This  visualization  technique 
is  better  than  traditional  graphics  primi¬ 
tives  (arrows)  in  high-resolution  models 
because  it  offers  less  visual  congestion. 
In  addition,  the  color  wheel  technique 
more  effectively  displays  eddy  forma¬ 
tion  and  vector  magnitudes  in  circula¬ 
tion  models. 

Two  major  changes  to  the  initial  color 
wheel  technique,  developed  at 
Mississippi  State  University  as  a  compo¬ 
nent  of  the  H  igh  Performance 
Computing  Management  Office 
Programming  Environment  and 
Training  Program  working  within  the 
Climate,  Weather,  and  Ocean  Modeling 
arena,  are  used  in  POPFlow  (an  inter¬ 
active  graphical  viewer  of  Parallel 
Ocean  Program  (POP)  modeling  data) 
to  achieve  interactive  graphics  perform¬ 
ance.  First,  the  Alpha  channel  is  linked 
to  the  vector  magnitude.  N  ormalizing 
vector  magnitude  gives  a  direct  correla¬ 
tion  to  transparency  or  opacity.  As  the 


magnitude  of  the  vector  approaches  the 
maximum,  the  pixel  contains  more 
color,  and  vice  versa.  Second,  a  simpler 
color  lookup  table  is  generated.  The 
FI  SV  colorspace  complicates  calcula¬ 
tions  with  converting  the  saturation  and 
value  components  to  the  Red-G  reen- 
Blue  (RGB)  colorspace,  the  de  facto 
colorspace  in  OpenGL.  The  hue  com¬ 
ponent  has  360 
degrees  directly  cor¬ 
related  to  direction 
(i.e.,  red  is  0  degrees 
and  is  due  east). 
Previous  color  wheel 
techniques  either 
converted  the  HSV 
to  RG  B  or  used  sim¬ 
pler  color  wheel 
lookup  tables.  The 
POPFlow  version 
uses  360-color 
entries  converted 
from  the  HSV  color- 
space.  POPFlow  is 
builton  the  OpenGL 
and  GLUT  applica¬ 
tion  programming 
interfaces  (API)  with 


hardware  alpha  and  blending.  The 
POPFlow  application  using  the  APIs, 
and  the  color  wheel  optimizations  easily 
show  a  year's  worth  of  vector  data, 
allowing  the  user  to  interact  with  ease. 
Color  wheel  optimizations  in  POPFlow 
show  impressive  global  animations  of 
major  effects  like  the  Gulf  Stream  cur¬ 
rents,  the  Kuroshio  currents  off  the  east¬ 
ern  coast  of  Japan,  and  equatorial  sta¬ 
tions.  Figure  1  is  a  global  image  of  the 
POP  model  using  the  original  color 
wheel  method.  Figure  2  shows  the 
enhanced  color  wheel  technique  in  the 
POPFlow  application.  Notice  that  in 
POPFlow,  the  alpha  blending  of  the 
color  wheel  layer  does  not  occlude  the 
bathymetry  below.  In  addition, 

POPFlow  has  interactive  adjustments 
for  vector  magnitudes  to  allow  users  to 
set  threshold  wheels  that  serve  to  filter 
these  magnitudes. 

Future  efforts  will  include  dynamic  color 
map  manipulation  and  incorporation  of 
a  similar  technique  to  show  convection 
in  fluid  flow  models.  For  more 
information  please  contact  the  NAVO 
MSRC  Visualization  Center  staff  at 
viz@navo.hpc.mil. 
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Figure  2.  Demonstration  of  the  enhanced  color 
wheel  technique  used  in  the  POPFlow  application. 
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The  vision  for  Navy  meteorological  and  oceano¬ 
graphic  forecasting  is  that  of  a  high-resolution 
global  coupled  air/ocean/ice  prediction  system 
with  very-high-resolution  regional  air/ocean  cou¬ 
pled  models  nested  into  the  global  system  at  key 
locations.  The  development  of  such  a  coupled 
prediction  system  is  a  computational  Challenge 
problem  reguiring  commensurate  supercomput¬ 
ing  resources.  Towards  this  goal  the  authors  are 
spinning  up  an  eddy-resolving  global  ocean 
model;  upon  completion  it  will  be  delivered  to 
the  Fleet  N umerical  Meteorology  and 
Oceanography  Center  (FLEN  U  M  M  ETOCCEN ) 
for  future  transition  to  the  operational  environ¬ 
ment. 

Any  ocean  model  to  be  used  in  such  a  prediction 
system  must  be  capable  of  simulating  high-fre¬ 
quency  (days  to  several  months)  and  short-scale 
(10  to  1000  km)  processes.  In  particular,  it  must 
be  capable  of  producing  realistic  mean  and  vary¬ 
ing  surface  and  thermocline  flows.  H  igh  horizon¬ 
tal  and  vertical  resolutions  are  reguired  to  pro¬ 
duce  not  only  mesoscale  variability,  but  also  the 
characteristics  of  strong,  narrow  currents  such  as 
the  Gulf  Stream.  Additionally,  the  model  must  be 
capable  of  producing  realistically  varying  thermo- 
haline  structures  in  the  upper  water  column  and 
therefore  reguires  a  mixed-layer  parameterization. 
Satisfying  these  reguirements  is  computationally 
demanding;  for  this  simulation  we  have  a 
Department  of  Defense  (DoD)  H  igh  Performance 
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Computing  Modernization  Office  (FI PCMO) 
Challenge  Grant.  Its  execution  is  taking  place  on 
the  Naval  Oceanographic  Office's  RS  16000  SP3 
(HABU). 

Ocean  Model  Description 

The  ocean  model  used  in  this  study  is  the  Los 
Alamos  National  Laboratory  (LAN L)  Parallel 
Ocean  Program  (POP).  It  is  a  primitive  eguation 
z-level  model  with  a  tree-surface  boundary  condi¬ 
tion.  Approximations  to  governing  fluid  dynamics 
eguations  permit  a  decoupling  of  the  model  solu¬ 
tion  into  barotropic  (vertically  averaged)  and 
baroclinic  (deviations  from  vertically  averaged) 
components;  these  are  solved  using  an  implicit 
elliptic  scheme  and  an  explicit  parabolic  eguation 
system,  respectively.  It  is  written  in  Fortran90  and 
is  designed  to  run  on  multi-processor  machines 
using  domain  decomposition  in  latitude  and  lon¬ 
gitude.  Message  Passing  Interface  (M PI)  is  used 
for  inter-processor  communications  on  distributed 
memory  machines  and  Shared  Message  Interface 
(SH  MEM)  on  shared  memory  machines. 
Benchmarking  shows  the  code  to  be  highly  scala¬ 
ble  onto  a  large  number  of  processors,  provided 
the  processor  sub-grid  is  large  enough.1  Further 
technical  details  and  references  regarding  the 
code  and  its  adaptation  for  massively  parallel 
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The  Ocean 


computers  can  be  obtained  from 
http://climate.acl.lanl.gov. 

A  global  0.1-degree,  40-level  configuration  of 
POP  is  being  spun-up  on  a  displaced  pole  grid 
(See  Figure  1),  whereby  the  North  Pole  is  rotated 
into  H  udson  Bay  to  avoid  a  polar  singularity.  The 
grid  consists  of  3600x2400x40  grid  points,  and 
the  spacing  is  about  11  kilometers  (km)  at  the 
equator,  decreasing  to  about  3  km  in  the  Arctic 
Ocean.  At  mid-latitudes  this  spacing  is  5-7  km. 

A  blended  bathymetry  was  created  from  Smith 
and  Sandwell,2  International  Bathymetric  Chart 
of  the  Arctic  Ocean  (IBCAO),3  and  British 
Antarctic  Survey  (BEDMAP)4  products.  All 
important  channels  and  sills  were  checked  and 
modified  to  facilitate  correct  flow. 

The  model  was  initialized  from  an  ocean  state 
depicted  by  temperature  and  salinity  from  the 
Navy's  1/8-degree  J  anuary  Modular  Ocean  Data 
Assimilation  System  (MODAS)  climatology  out¬ 
side  of  the  Arctic  and  the  University  of 
Washington's  Polar  Hydrography  winter  clima¬ 
tology  in  the  Arctic  (http://psc.apl. 
washington.edu/Climatology.html).  To  produce 


an  energetically  realistic  ocean,  synoptic  surface 
forcing  is  used  whenever  possible  during  a  20- 
year  adjustment  of  the  model  to  its  initial  state. 
This  forcing  was  largely  constructed  from 
National  Center  for  Environmental  Prediction 
(NCEP)  fluxes  for  1979-1998. 5  A  mixed  layer  for¬ 
mulation  is  active.6  To  date,  about  a  decade  of 
the  simulation  has  been  completed. 

The  Global  Simulation 

The  realistic  simulation  of  sea  surface  tempera¬ 
ture  (SST)  will  be  important  both  to  the  predic¬ 
tion  of  fronts  and  eddies  and  for  coupling  to  the 
ice/atmosphere  components  of  the  future  predic¬ 
tion  system.  A  snapshot  of  temperature  in  the 
uppermost  model  level  on  1  February  1987  is 
seen  in  Figure  2.  To  display  the  whole  globe  at  the 
full  model  resolution,  the  field  is  plotted  in  model 
coordinates;  hence,  the  distortion  in  the  northern 
hemisphere.  Well-defined  frontal  structures  a ssoj 
ated  with  the  western  boundary  currents 

Article  Continues. 
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Figure  7.  Spacing  (km)  of  0.1°,  40-level  global 
POP  grid. 
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and  tropical  instability  waves  are  observed. 
The  large  meander  of  the  Kuroshio  Current 
seen  to  the  south  of  J  apan  is  one  of  its 
known  meander  states  at  this  location; 
movie  loops  indicate  that  the  model  also 
reproduces  the  other  observed  states. 

M  esoscale  features  are  apparent  in  all 
ocean  basins;  particularly  striking  are  the 
Agulhas  Current  eddies  that  are  shed  south 
of  Africa. 

To  more  fully  appreciate  the  mesoscale 
activity  in  the  model,  the  flow  in  the  upper 
level  of  the  model  is  seen  in  Figure  3.  H  ere 
the  direction  and  speed  of  the  flow  are 
represented  by  color  and  hue,  respectively. 
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Figure  2.  Snapshot  of  temperature  in  the  uppermost  model  level 
(5  m)  on  15  J  anuary  1987. 


To  understand  the  interactions  between  the 
large-scale  flows  of  the  Western  Pacific,  the 
Asian  marginal  seas,  and  the  Indonesian 
Throughflow,  a  snapshot  of  surface  velocity 
from  1 J  une  1987  is  displayed  (See  Figure 
4).  This  is  a  challenging  region  for  an  ocean 
model  to  simulate  realistically,  and  is  there¬ 
fore  a  good  test  of  the  model's  capabilities. 
Again,  direction  and  strength  are  indicated 
by  color  and  color  intensity,  respectively. 

G  rid  distortion  is  apparent  in  the  northern 
part  of  the  domain,  particularly  in  the 
Kuroshio  Extension.  The  flow  in  this  region 
is  complex  and  punctuated  by  mesoscale 
activity;  regardless,  details  are  in  gualitative 
agreement  with  observations.7 

The  N orth  Eguatorial  Current  (NEC),  iden¬ 
tified  by  the  northwestward  (blue)  flow  off¬ 
shore  of  the  Philippines,  bifurcates  near  the 


Figure  3.  Snapshot  of  surface  velocity  from  the  uppermost 
model  level  (5  m)  on  1  J  anuary  1983.  Color  and  hue  represent 
current  direction  and  intensity,  respectively.  The  expanded 
region  demonstrates  eddy  shedding  by  the  Agulhas 
Retroflection. 


The  Agulhas  Retroflection  Current  system  is  enlarged  to 
highlight  the  shedding  of  eddies,  their  sense  of  rotation, 
and  their  northwest  movement  across  the  South  Atlantic 
Ocean.  The  characteristic  reversing  zonal  flows  of  the 
tropics  can  be  easily  identified  along  with  the  associated 
standing  meander  patterns  of  the  N  orth  Eguatorial 
Countercurrents  in  the  western  Atlantic  and  Pacific 
oceans.  Rich  eddy  fields  are  associated  with  the  other 
boundary  flows  and  the  Antarctic  Circumpolar  Current 
(ACC)  in  the  Southern  Ocean. 
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coast  to  flow  northwestward  as  the 
Philippines  Current  and  southward 
as  the  M  indanao  Current.  The 
Kuroshio  is  identified  as  the  strong 
northeastward  current  off  Taiwan;  to 
the  north  an  intrusion  of  the  current 
is  seen  in  the  East  China  Sea.  The 
Tsushima  Current  enters  the  J  apan 
Sea  and  continues  in  a  northeast¬ 
ward  direction  along  the  coast  of 
Japan.  The  Mindanao  Current  turns 
southeastward  to  enter  the  Celebes 
Sea,  and  southward  flow  is  seen 
between  Borneo  and  Sulawesi.  Flow 
into  the  Indian  Ocean  is  seen 
through  all  the  Indonesian  straits  to 
the  south  of  these  islands.  The  circu¬ 
lation  in  this  region  is  dominated  by 
the  seasonal  monsoons,  and  south¬ 
ward  flow  into  the  Indian  Ocean  is 
consistent  with  observations  at  this 
time  of  year.8  In  February,  the  sur¬ 
face  flow  reverses,  and  the  net  flow 
is  away  from  the  Indian  Ocean.  The 
Indonesian  Throughflow  is  the  major 
low-latitude  conduit  of  properties 
between  the  Indian  and  Pacific 
oceans  and  is  therefore  important  on 
both  short-term  and  climatic  time 
scales.  Quantitative  evaluations  and 
analyses  of  the  spin-up  are  under¬ 


way  using  data  sets  with  global  or 
near-global  coverage  such  as  those 
obtained  from  satellite  altimeters, 
surface  drifting  buoys,  and  vertical 
profilers.  We  are  comparing  modeled 
and  observed  mean  flows,  transports 
through  key  passages,  energy  levels, 
intrinsic  scales,  and  mixed  layer  vari¬ 
ability,  among  others. 

In  this  manner  we  are  able  to  assess 
the  progress  of  the  spin-up  and 


make  adjustments  as  appropriate. 
Following  the  20-year  spin-up,  a 
decade-long  simulation  starting  in 
the  1990s  will  be  performed  using 
high-frequency  Navy  Operational 
Global  Atmospheric  Prediction 
System  (NOGAPS)  surface  forcing. 
Further  details  of  the  model  run, 
benchmarking,  results,  and  movie 
loops  of  the  spin-up  are  available  at 
http://www.oc.nps.navy.mil/navypop. 


Figure  4.  Snapshot  of  surface  flow  on  1 J  une  1987  in  the  Western 
Pacific,  the  Asian  marginal  seas,  and  the  Indonesian  Throughflow. 
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VSIPL/ERI:  An  Enhanced  Reference  Implementation 
of  the  Vector  Signal  and  Image  Processing  Standard 

Gary  Boudreaux  and  Dr.  Tony  Skjellum,  MPI  Software  Technology,  Inc. 


As  a  result  of  the  N  AVO  M  RSC 
Programming  and  Environmental 
Training  Program  (PET)  Tiger  Team 
Collaboration  Final  Report,1  the  cre¬ 
ation  of  a  new  performance  Vector 
Signal  and  Image  Processing  Library 
(VSIPL)2  that  interfaced  with  opti¬ 
mized  signal  processing  and  linear 
algebra  libraries  was  assigned  by 
PET  to  MPI  Software  Technology  Inc. 
(MSTI).  The  goal  of  the  four-month 
project  was  to  begin  with  the  Tactical 
Advanced  Signal  Processing  (TASP) 
reference  implementation  of  VSIPL 
and  to  "wrap"  function  calls  to  the 
Linear  Algebra  Package  (LAPACK), 
Basic  Linear  Algebra  Subroutines 
(BLAS),  and  Fastest  Fourier 
Transform  in  the  West  (FFTW)  opti¬ 
mized  libraries.  This  new  software 
was  entitled  VSIPL  Enhanced 
Reference  Implementation 
(VSIPL/ERI). 

Background 

VSIPL,  a  community  standard,  con¬ 
solidated  and  streamlined  existing 
mathematical  libraries  and  defined 
standard  functions  for  the  scientific 
and  engineering  community.  VSIPL 
functions  include  Linear  Algebra, 
Signal  Processing  (i.e.,  Fast  Fourier 
Transform  (FFT)  Filters),  and  Scalar, 
Vector,  and  Matrix  functionality.  The 
main  features  are  its  portability, 
object-based  description,  and  run¬ 
time  modes  for  both  development 
and  production.  The  library  also  fea¬ 
tures  opaque  objects  such  as  blocks, 
views  on  the  blocks  (i.e.,  vectors, 
matrices,  and  tensors),  explicit  mem¬ 
ory/algorithm  hints,  and  public  and 
private  data  arrays.  As  an  active 
member  of  the  VSIPL  forum,  MSTI 
ensures  that  its  current  implementa¬ 
tions  of  the  libraries  conform  to  the 
VSIPL  standard. 


As  a  baseline  of  what  is  available  for 
embedded  computing,  MSTI's  com¬ 
mercial  implementation  ofVSPIL 
uses  altivec  and  assembly  language 
optimizations  to  achieve  high  per¬ 
formance  on  element  wise  and  signal 
processing  functions.  It  contains 
advanced  Ct+  programming 


concepts,  and  the  source  is  actually  a 
C  +  +  implementation  with  C  APIs. 
MSTI  supports  the  VSIPL  Core 
Profile  for  several  operating  systems 
and  the  PowerPC  (G 4)  and  Intel 
(Pill)  processors.  Flowever,  such 
software  has  not,  as  yet,  been  made 
available  for  the  M  SRC  s'  current 
platforms. 

Thus,  an  interim  solution  for  higher 
performance  VSIPL  is  required. 
VSIPL/ERI3  satisfies  this  requirement. 
In  particular,  VSIPL/ERI  is  freely 
available  middleware  for  United 
States  (U.S.)  Government 
Department  of  Defense  (DoD)  M  SRC 
sites.  The  project  requirements  were 


to  enhance  the  performance  of  the 
TASP4  VSIPL  reference  implementa¬ 
tion  by  using  commercial-off-the- 
shelf  (COTS)  lower  level  libraries, 
instead  of  the  TASP  kernels. 

This  was  accomplished  by  replacing 
sections  of  the  TASP  VSIPL  reference 
implementation  with  wrappers  to 


BLAS,  LAPACK,  and  FFTW.  Such 
libraries  are  available  on  many  sites. 
This  work  involved  the  identification 
of  the  appropriate  functions  in  the 
libraries  to  wrap,  the  creation  and 
extraction  of  pointers  to  the  appro¬ 
priate  data  from  the  VSIPL  data 
structures,  compliance  with  VSIPL 
data  alignment  restrictions  (with 
regards  to  memory  management), 
and  the  resolution  of  problems  asso¬ 
ciated  with  the  C-to-Fortran  inter¬ 
face.  The  project  required  the  port  of 
the  VSIPL/ERI  software  to  four  plat¬ 
forms:  IBM  AIX  (Power  3); 
SOLARIS/SPARC;  Linux  (x86);  and 
MIPS  (SGI,  R10K).  Currently,  the 
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library  has  been  delivered 
for  SPARC,  Linux,  and  AIX. 
M  STI  has  been  granted 
access  to  the  NAVO  SGI 
machine,  and  the  entire 
project  was  completed 
before  15  September  2001. 

Related  Technologies 

The  VSIPL/ERI  library  uti¬ 
lizes  the  FFTW  library  for  all 
FFT  functions,  LAPACK 
library  for  solving  advanced 
systems  of  linear  equations, 
and  the  BLAS  library  for 
basic  linear  algebra  to 
enable  it  to  exceed  TASP 
VISPL  performance. 

FFTW4,  developed  by  Dr. 
Matteo  Frigo  and  Steven  G. 

J  ohnson  at  the 
Massachusetts  Institute  of 
Technology  (M  IT),  is  a  C 
subroutine  library  for  com¬ 
puting  the  Discrete  Fourier 
Transform  (DFT)  in  one  or 
more  dimensions,  of  both 
real  and  complex  data,  and 
of  arbitrary  input  size.  The 
library  is  known  for  its  fast 
performance  and  excellent 
portability  between  architec¬ 
tures.  The  latest  official  ver¬ 
sion  is  2.1.3,  and  in  1999 
FFTW  received  the  J .  H  . 
Wilkinson  Prize  for 
Numerical  Software.5 

In  addition  to  FFTW, 
VSIPL/ERI  employs 
LAPACK5  routines.  As  a 
Fortran-77  subroutine 


^giire  3  Cqfrfls'Hiitir  i  Inear AJpjfra  Benchmarks 
ESI 

■JD-  ■’fl'IMHi  Pll  'Hn  2MV  ra-n 

-uncficn  i  “"A1 1  1  iMh.^  nMinM-jmss” 

1 

lllpLUSi 

M-123 .h>64.K-=32 

CCiaisakv  ■ 

2 

H1>!.6 

’1  56 

Cholc3ky r 

1 

iti  7S 

2.1 

jluO  r 

1 

rD 

21  A 

LUQJ 

2 

*0  21 

SS'n  p.  T 

"  l  ! 

- 7771 

1?i~V 

IGdiTin  1 

1 

j:31D 

E2.S 

SpntCr 

i 

•iivptodj 

2 

C  J11 

0  J1 

CMVpind.r 

? 

1 

QJ66 

tor  7 

2 

1,47 

EtS 

cyouterj 

3 

1  96 

D.3. 

a 

- i T7T- 

- w 

GOaVFkSM 

j 

ifis  96 

20  in 

■4-sqj 

2 

a  i:  as 

1^195  5J7 

QKCj" 

2 

“TT5 

i.&j 

raflb  ■ 

i 

54  06 

runcdonj 

#ASP 

siz* 

'■■p-ftej 

•!=2St?.h=  I2o.K=G' 

fcfciss  K=32 

^£rioicsk>  F 

"  "S™ 

16  7  J 

Chileskyj 

1 

77  D 

7.3 

cluc  r 

1 

3=7* 

7S  i-i 

r.Li  f  : 

■T' 

TT35 

Girt  fi r 

1 

IE  S71 

I-GC  TiP  f 

1 

'.-IOC 

ESJ  t- 

tfppdiS  f 

1 

am 

“20  94 

■vWjTMrrU 

U 

 41 

on 

CttVpradj" 

2 

1  C’E- 

D.29 

VtutarJ 

r > 

■li. 

- j? 

0  13 

iTV3-.l»r  r 

■  ?.® 

~  P"£?| 

rJOVSQl r 

3 

77  5 

id  2: 

i'Citayar.  r 

- 

- w? 

L SQ  T 

3 

5  Lfi  7E  57 

■  9  96 

t»D.l 

7  7  ! 

^a“RD,f 

~~7m 

7tfSi 

library,  the  well-known 
LAPACK  Application 
Programming  Interface 
(API)  routines  provide 
solutions  to  systems  of 
simultaneous  linear  equa¬ 
tions,  least-squares  solu¬ 
tions  of  linear  systems  of 
equations,  eigenvalue 
problems,  and  singular 
value  problems.  In  addi¬ 
tion,  LAPACK  also 
includes  other  matrix  fac¬ 
torizations  such  as  Logical 
Unit  (LU),  Cholesky,  QR, 
and  SVD,  and  its  library  is 
equipped  to  handle  both 
dense  and  banded  matri¬ 
ces. 

For  all  routines,  LAPACK 
provides  functionality  for 
real  and  complex  matrices 
in  both  single  and  double 
precision.  These  routines 
are  written  so  that  the 
maximum  amount  of  com¬ 
putation  is  handled 
through  calls  to  the  BLAS. 
LAPACK  was  designed  to 
exploit  the  Level-3  BLAS6 
—  a  set  of  specifications 
for  Fortran  subprograms 
that  perform  various  types 
of  matrix  multiplication 
and  the  solution  of  triangu¬ 
lar  systems  with  multiples 
right-hand  sides.  The 
coarse  granularity  of  the 
Level-3  BLAS  operations 
causes  it  to  promote  high 
efficiency  on  many  high- 

Article  Continues  Page  26... 
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NAVO  MSRC  PET  Update 

Eleanor  Schroeder,  NAVO  MSRC  Programming  Environment  and  Training  Program 
(PET)  Government  Lead 

The  new  PET  program  is  approaching  the  end  of  its  first  year. 

As  with  any  new  start,  it  has  been  a  tremendous  learning  expe¬ 
rience  for  all  involved. 

At  the  NAVO  MSRC,  as  reported  in  the  last  issue  of  the 
Navigator ,  we  are  primarily  responsible  for  3  of  15  PET 
Functional  Areas  (FAs).  Those  three  areas  are  the  two 
Computational  Technology  Areas  (CTAs)  of  Climate,  Weather, 
and  Oceanography  Modeling  (CWO)  and  Environmental 
Quality  Modeling  and  Simulation  (EQM).  The  NAVO  MSRC 
PET  program  is  also  responsible  for  taking  the  lead  on  the 
cross-cutting  functional  area  of  Computational  Environments 
(CEs).  More  information  about  these  functional  areas  can  be 
found  on  the  El  igh  Performance  Computing  M  odernization 
Program  Office  (EH  PC  M  0 )  web  site  under  the  PET  link. 

M  ost  of  our  positions  have  been  filled  or  are  in  the  process  of 
being  filled.  We  have  had  a  few  changes,  so  the  following  is  a 
who's  who  at  the  NAVO  MSRC  PET  program: 

Government  Lead:  Eleanor  Schroeder 

Component  Point  of  Contact  (POC):  Dr.  George  Heburn 
A  Executive  Assistant:  Tammy  Smith 

^ Technologists:  Andrew  Schatzle  and  Brian  Tabor 

CWO  FA  POC:  Dr.  J  ay  Boisseau 

^CWO  On-site  (NAVO  MSRC):  Dr.  Tim  Campbell 

^  CWO  On-site  US  Army  Engineer  Research  and 
Development  Center  (ERDC):  Dr.  Phu  Luong 

AC\NO  On-site  (Monterey):  Applicant  being  inter¬ 
viewed 

EQM  FA  POC:  Dr.  Mary  Wheeler 

A  Environmental  Quality  Modeling  and  Simulation 
(EQM)  On-site  (ERDC):  Drs.  Lea  J  enkins  and 
Victor  Parr  (interim) 

A  EQM  On-site  (NAVO  MSRC):  Position  to  be  filled 
CE  FA  POC:  Dr.  Shirley  Moore 
CE  On-site  (NAVO  MSRC):  Dr.  Tom  Cortese 

The  three  FA  POCs  have  submitted  the  PET  strategies  for 
each  of  their  functional  areas.  Those  strategies  can  be  found 
in  the  sidebar  to  this  article. 

One  of  the  highlights  of  PET  is  the  refurbishing  (many  thanks  to 
the  NAVO  MSRC)  of  the  equipment  in  our  PET  classrooms.  We 
now  have  20  new  single-processor  Pentium  4  workstations  oper¬ 
ating  at  1.8  gigahertz  with  1  gigabyte  of  memory  and  a  20-giga- 
byte  removable  hard  drive,  which  gives  us  the  capability  of  host¬ 
ing  either  Linux-based  or  Windows-based  classes.  The  equip¬ 
ment  was  given  its  test  run  at  the  H  PC  MO-sponsored 
Service/Agency  Approval  Authority  meeting  that  we  hosted  here 
in  February. 

One  final  note,  PET  will  undergo  its  first  review  in  May  2002  in 
Vicksburg,  MS. 


CWO  Strategy 

Dr.  J  ay  Boisseau,  University  of  Texas,  Austin, 
CWO  FA  POC 

The  MOS  team  supports  the  DoD  climate/weather/ocean 
(CWO)  modeling  community  in  the  new  PET  program.  The 
CWO  support  strategy  was  developed  by  the  MOS  team  in  the 
PET  recompetition  and  has  been  refined  during  the  program 
with  input  from  many  DoD  users.  The  MOS  team  for  CWO 
includes: 

□  George  Fleburn  (Mississippi  State  University  (MSU)), 
Component  Lead  for  the  MOS  team  overseeing  CWO, 
EQM,  and  CE 

□  J  ay  Boisseau  (Texas  Advanced  Computing  Center 
(TACC )/U  niversity  of  Texas  (UT)  at  Austin),  coordinator 
for  MOS  CWO  support  activities 

□  Tim  Campbell  (MSU)  and  Phu  Luong  (ERDC),  on-site 
CWO  support  staff 

□  DoDCWOUser  Advisory  Panel:  Alan  Wallcraft  (N  RL- 
Stennis),  J  ane  Smith  (ERDC),  Rich  H odur  (N  RL- 
Monterey),  and  Frank  Ruggiero  (AFRL),  plus  the  CTA 
leader  for  CWO,  Bill  Burnett  (CN  MOC) 

The  PET  CWO  support  strategy  is  focused  on  the  develop¬ 
ment,  implementation,  and  support  of  advanced  computing 
technologies  that  are  needed  to  enhance  the  current  and  next 
generations  of  strategic  CWO  applications.  Since  the  DoD 
CWO  community  is  among  the  world  leaders  in  atmospheric 
and  ocean  physics,  the  PET  CWO  support  strategy  is  focused 
on  advanced  computing  technologies.  M  any  of  these  technolo¬ 
gies  are  critical  for  enhancing  the  performance,  scalability,  and 
modeling  capabilities  of  strategic  CWO  applications  and  for 
analyzing  the  resulting  data.  The  MOS  team  supports  the 
development  and  direct  application  of  these  technologies  to 
CWO  research  problems  and  the  integration  of  these  tools  into 
CWO  codesand  research  infrastructure.  Some  particular  activ¬ 
ities  requested  by  DoD  users  include:  (1)  porting  codes  to  the 
newest  architectures  and  optimizing  them  for  maximum  per¬ 
formance  and  scalability,  (2)  evaluating  various  grid/mesh 
techniques  (unstructured  grids  for  coastline,  different  vertical 
coordinate  systems)  for  accuracy  and  performance,  and  (3) 
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developing  and  implementing  code  coupling  techniques 
for  ocean  codes  with  atmospheric  and/or  ice  models,  but 
also  for  integrating  other  kinds  of  models  (aerosol  trans¬ 
port,  biology/chemistry,  etc.) 

CE  Strategy 

Dr.  Shirley  Moore,  University  of  Tennessee, 
Knoxville,  CE  FA  POC 

The  CE  team  is  led  by  the  Innovative  Computing 
Laboratory  (ICL)  at  the  University  of  Tennessee-Knoxville 
(UTK).  ICL,  under  the  leadership  of  Distinguished 
Professor  J  ack  Dongarra,  is  a  world-recognized  leader  in 
many  areas  of  high  performance  computing  (H  PC).  The 
CE  FA  POC  is  Dr.  Shirley  M  oore,  who  is  the  Associate 
Director  of  Research  for  ICL.  Dr.  Moore  is  assisted  by  Dr. 
David  Crank,  a  research  scientist  with  ICL,  and  by  Dr. 
Thomas  Cortese,  the  onsite  CE  lead  at  NAVO  MSRC. 

Other  highly  qualified  personnel  at  UTK  and  other  PET 
team  institutions  are  involved  in  CE  projects. 

The  CE  vision  is  to  improve  user  productivity  and  applica¬ 
tion  performance  through  the  provision  of  effective  parallel 
programming  tools  as  part  of  a  consistent  well-document¬ 
ed  CE.  Achieving  this  goal  involves  a  number  of  activities. 
CE  staff  are  collaborating  with  Shared  Resource  Center 
(SRC)  systems  staff  to  implement  a  CE  across  the  SRCs 
that  is  as  consistent  as  possible.  The  CE  includes  compil¬ 
ers,  message-passing  libraries,  debugging  and  performance 
analysis  tools,  and  data  management  and  visualization 
tools.  CE  staff  are  working  to  ensure  that  all  components 
of  the  environment  are  operating  correctly  in  the  SRC 
batch-queuing  environments  and  that  they  are  adequately 
supported  and  documented.  Cross-platform  tools  are  being 
made  available  wherever  possible  so  that  users  do  not 
need  to  learn  a  different  tool  interface  for  each  platform. 
These  tools  include  the  TotalView  debugger,  Vampir 
Message  Passing  Interface  (M  PI)  performance  analysis  tool, 
and  K  A  P/Pro  Toolset  for  OpenM  P  programming,  all  of 
which  are  commercial  tools.  Also  included  is  the  cross-plat¬ 
form  library  Performance  Application  Programming 
Interface  (PAPI)  to  hardware  performance  counters,  as  well 
as  freely  available  performance  analysis  tools  for  sites  and 
users  who  do  not  have  access  to  commercial  tools.  The 
environment  will  be  updated  regularly  to  keep  pace  with 
changes  in  architectures  and  operating  systems. 

CE  core  support  consists  of  ongoing  activities  in  the  areas 
of  assessing  and  meeting  user  needs  and  delivering  effec¬ 
tive  training  in  El  PC  architectures  and  parallel  program¬ 
ming  tools.  Core  support  activities  include  organization  of 
a  CE  user  advisory  panel  to  help  determine  CE  strategic 
goals  and  user  requirements  and  to  help  with  evaluation  of 
CE  tools,  documentation,  and  training  materials.  Core  sup¬ 
port  personnel  are  also  involved  in  evaluation  and  beta¬ 
testing  of  new  tools  and  new  versions  of  existing  tools.  The 


CE  training  strategy  involves  development  of  a  compre¬ 
hensive  training  curriculum  to  give  DoD  users  the  knowl¬ 
edge  and  skills  they  need  to  effectively  use  El  PC  platforms 
at  the  SRCs.  Training  topics  include  specific  programming 
languages,  parallel  programming  models,  performance 
optimization  techniques,  and  debugging  and  performance 
analysis  tools.  Another  CE  activity  is  to  anticipate  and  plan 
for  CE  needs  in  the  context  of  emerging  El  PC  trends  and 
systems,  such  as  cluster  computing,  meta-computing,  and 
interactive  and  real-time  requirements. 

CE  projects  target  specific  areas  where  critical  user  needs 
have  been  identified.  Areas  that  have  been  identified  by 
the  CE  user  advisory  panel  include  application  portability, 
application  performance  evaluation  and  tuning,  a  consis¬ 
tent  well-documented  CE,  data  management,  and  parallel 
and  asynchronous  input/output.  Current  CE  projects  are 
addressing  a  consistent  CE  and  deployment  of  PAPI  for 
the  purpose  of  application  performance  analysis.  Project 
proposals  to  address  these  and  the  other  newly  identified 
needs  are  currently  being  formulated. 

EQM  Strategy 

Dr.  Mary  Wheeler,  University  of  Texas,  Austin, 

EQM  FA  POC 

The  EQM  FA  under  PET  has  defined  three  primary  strate¬ 
gic  efforts.  The  first  effort  involves  the  development  and 
implementation  of  new,  accurate  parallel  discretizations 
with  adaptivity  that  will  be  based  on  a  posteriori  error  esti¬ 
mators.  In  addition,  this  effort  will  include  robust  scalable 
parallel  solvers. 

A  second  effort  involves  code  couplings  of  different  physi¬ 
cal  processes  (e.g.,  flow,  transport,  reactions,  and  mechan¬ 
ics)  which  occur  within  the  same  physical  domain,  or  when 
different  physical  regimes  (e.g.,  surface/subsurface,  and 
fluid/structure)  interact  through  interfaces.  The  issues  to  be 
considered  include  (1)  appropriate  physical  quantities  or 
transmission  conditions  that  are  dependent  on  the  applica¬ 
tion  for  coupling  across  physical  domains,  (2)  development 
of  algorithms  for  modeling  transmission  conditions,  (3) 
time-stepping  selection  for  treating  couplings  of  different 
physical  models,  (4)  grid  projection  strategies,  and  (5)  com¬ 
puter  science  tools  for  implementing  coupled  processes. 

The  third  effort  involves  the  integration  of  emerging  com¬ 
putational  tools  for  handling  large  data  sets,  grid/metacom¬ 
puting,  interactive  steering,  scientific  visualization,  mesh 
partitioning,  and  dynamic  adaptive  grids. 

It  is  anticipated  that  the  algorithms  and  tools  developed 
under  these  efforts  will  be  applied  to  EQM/CWO  software 
such  as  Adaptive  EH  yd rau lies  (AD H  ),  CE-QUAL-ICM , 
ADCIRC,  and  CH 3D. 
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Steve  Adamec, 
Director,  NAVO 
MSRC,  briefs  visit¬ 
ing  members  of 
the  U.S. 
Commission  on 
Ocean  Policy. 


^F  NAVOCEANO 

Change  of 
Command 
^F  Ceremony  8 
^F  March  2002. 

^F  Participants  L-R, 
^F  CAPT  Philip  G. 

Renaud  (incoming), 
f  RDM L  Thomas  Q. 
Donaldson,  V, 

Commander,  NMOC, 
and  CAPT  Timothy 
McGee 


Left: 

RDM L  Thomas  Q. 
Donaldson,  V,  and 
members  of 
the  U.S.  Commission 
on  Ocean  Policy. 


Right: 

(L-R)  CDR  Raul  Martinez  Sanchez, 
Director  of  Hydrographic 
Department,  Hydrographic  and 
Cartographic  Office  (DIGADHICAR), 
LCDR  Rafael  Ponce,  Director  of 
Tides  and  Maritime  Publications, 
DIGADHICAR,  and  RADM  Anastacio  F. 
DeAbiega  Gamez,  Director  General, 
DIGADHICAR,  Eric  Villalobos, 
NAVOCEANO  International  Division. 


Left: 

(L-R)  CDR  Augusto 
Mourao  Ezequiel, 
Technical  Director, 
Portuguese  Hydrographic 
Institute,  VADM  J  ose 
Torres  Sobral,  General 
Director,  Portuguese 
Hydrographic  Institute  and 
Dave  Cole,  NAVO  MSRC, 
Carol  Nichols,  CNMOC. 


Left: 

(L-R)  LCDR  Erhan  Gezgin, 
Project  Officer,  Turkish 
Navy  Department  of 
Navigation,  Hydrography 
and  Oceanography 
(TNDHNO),  CPT  AM 
Kaplan,  Head  of 
Technology  Group, 
TNDHNO,  CDR  Ahmet 
Turker,  Chief  of 
Oceanography  Division 
Michael  J  effries, 
NAVOCEANO 
I  nternational 
Division. 


Right: 

(L-R)  Dave  Cole,  NAVO 
MSRC,  CPT  Eric  Fraser, 
British  Naval  Staff, 
British  Embassy,  and 
LCDR  Don  Ventura. 


Right: 

Members  of  the  Basic 
Officer  Assession 
Training  course  at  the 
Naval  Meteorology  and 
Oceanography  Professional 
Development  Center. 


FT. 


- 


Members  of  the  editorial  and 
reporting  staff  of  The  Sun 
Herald  (Biloxi,  MS)  attending  a 
presentation  by  Pete 
Gruzinskas  of  the  NAVO 
MSRC  Visualization  Center. 


(L-R)  Frank 
Lovato,  NAVO 
MSRC  Security  1 
Officer,  Doug  Lee, 
Director,  Louisiana 
Research  and 
Technology 
Park  Baton  Rouge, 
and  companions 


(L-R,  facing)  LTC  J  eff  Pritchard 
(USAF),  LCDR  Dave  Manero 
(USN),  MAJ  J  im  Hecker  (USAF) 
of  Senator  Trent  Lott's  staff. 


Right:  i 

(L-R)  Dave  Cole,  LCDR 
Henry  Howell,  United 
Kingdom  (UK)  Personal 
Exchange  Program 
(PEP)  Officer,  FNMOC, 
and  Tom  Crew,  Allied 
Cooperation/ 
Technology  Transfer 
Division,  NAVOCEANO. 


Right: 

(L-R)  ADM  V.E.  Clark, 
Chief  of  Naval 
Operations  and  Steve 
Adamec,  Director 
NAVO  MSRC. 


Left: 

Visit  of  Steve  Hall,  National  Imagery 
and  Mapping  Agency  Gulf  of  Mexico 
director. 


Left: 

Members  of  the 
PET  MOS  Team  are 
familiarized  with  the 
NAVO  MSRC  by  Dave 
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Navigator  Tools  and  Tips 

File  Transfer  Techniques:  Transferring 
Between  HABU  and  Other  Systems 


Sheila  Carbonette,  NAVO  MSRC  User  Support 

This  article  describes  a  method  for  transferring  files 
between  HABU  and  other  systems  via  the  transfer  queue 
using  a  stream  of  job  steps  in  a  single  job  command  file. 
The  transfer  queue  was  created  as  a  shared  usage  queue 
for  users  to  transfer  input  and/or  output  files  before/after 
the  executable  program  as  serial  jobs. 

This  means  the  transfer  queue  shares  the  interactive  nodes 
with  other  users.  In  order  to  submit  a  job  to  this  queue, 
you  must  specify  the  following  in  the  job  step: 

#@  class  =  transfer 
#@  node_usage  =  shared 

To  specify  a  stream  of  job  steps,  you  need  to  list  each  job 
step  in  the  job  command  file.  Each  job  step  must  be  fol¬ 
lowed  by  the  "queue"  statement.  The  "dependency"  key¬ 
word  is  used  to  let  LoadLeveler  know  that  the  job  steps  are 
not  independent  and  that  the  next  job  will  run  based  on 
the  exit  status  of  the  previously  run  job.  In  the  example 
below,  there  are  three  job  steps: 

1.  A  serial  job  step  to  transfer  the  input  file  via  the  transfer 
queue. 

2.  A  parallel  job  step  to  run  a  sample  POE  script  via  the 
batch  queue. 

3.  A  serial  job  step  to  transfer  the  output  file  via  the  transfer 
queue. 

N  ote:  Each  job  step  must  have  a  unique  name  for  all  out¬ 
put  and  error  files. 

Example  job  command  file: 

fl8nl3e%  more  multistepjobscript 
#!/bin/ksh 

*****  STEP  1- SERIAL  ***** 
account_no=NA0101 
class  =  transfer 
step_name  =  stepl 
executable  =get_file.ksh 
input  =  /dev/null 

output  =  $(jobid).step$(stepid).log 
error  =  $(jobid).step$(stepid).log 
environment  =  C0PY_ALL 
initialdir  =  /scr/shecar/multisteps 
job_name  =  multistep 
job  type  =  serial 
node_usage  =  shared 
notification  =  never 
notify_user  =  shecar 
shell  =  /bin/ksh 
wall_clock_limit  =  300 
queue 

******  STEP  2 -PARALLEL***** 
account  no=NA0101 
class  =  batch 


dependency  =  (stepl  =0) 
step_name  =step2 
executable  =poe_script.ksh 
input  = /dev/null 

output  =  $(jobid).step$(stepid).log 

error  =  $(jobid).step$(stepid).log 

environment  =  COPY_ALL;  M P_LAB E LIO  =yes 

initialdir  = /scr/shecar/multisteps 

job_name  =  multistep 

job  type  =  parallel 

node  =  4 

tasks_per_node  =4 
node_usage  =not_shared 
notification  =  never 
notify_user  =  shecar 
shell  = /bin/ksh 
wall_clock_limit  =  600 
queue 

#  *****  j  o  B  STEP  3 -SERIAL***** 
account_no=NA0101 
class  =  transfer 
node_usage  =  shared 
dependency  =  (step2  =0) 
step_name  =step3 
executable  =put_file.ksh 
input  = /dev/null 

output  =  $(jobid).step$(stepid).log 
error  =  $(jobid).step$(stepid).log 
environment  =  COPY_ALL 
initialdir  =  /scr/shecar/multisteps 
job_name  =  multistep 
error  =  $(jobid).step$(stepid).log 
environment  =  COPY_ALL 
initialdir  =  /scr/shecar/multisteps 
job_name  =  multistep 
job  type  =  serial 
notification  =  always 
notify_user  =  shecar 
shell  = /bin/ksh 
wall_clock_limit  =  300 
queue 

******  END  OF  multistepjobscript  SCRIPT  ***** 


Job  Step  1  Executable 


fl8nl3e%  more  get  file.ksh 

#!/bin/ksh 

let  R  C  =0 

#***  IDENTIFY  NODE  RUNNING  SERIAL  J  OB  STEP  *** 
test  script. ksh 

#**t  REMOVE  FILE  FROM  PREVIOUS  RUN  *** 
rm  test  file 
let  RC  =$? 
if  (($RC  !=  0)) 

then  /bin/echo  "$0-ERROR:  Removing  test  file;  RC=$RC"  exit 
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When  this  job  command  file  is  submitted  to  LoadLeveler, 
three  jobs  will  be  placed  in  the  queue.  This  can  be  verified 
by  using  the  "llq"  command. 


fl8nl3e%  llq  |  egrep  "1|shecar" 


ID 

Owner 

Submitted 

ST 

PR 

Class 

fl8nl3s.416 

73.0 

shecar 

3/21  09:11 

1 

50 

transfer 

fl8nl3s.416 

73.2 

shecar 

3/21  09:11 

NQ 

50 

transfer 

fl8nl3s.416 

73.1 

shecar 

3/21  09:11 

NQ 

50 

batch 

Initially  the  first  job  is  queued,  and  the  second  two  will 
wait  until  the  previous  one  finishes. 

Job  Step  1  Output 


fl8nl3e%  more  41673.step0.log 
Hostname  =fl8nl3e 

Date  =Thu  Mar  21  09:37:16  CST  2002 

File  Staging  FROM  j  ules  Completed  SUCCESSFULLY! 


Job  Step  2  Output 


fl8nl3e%  more  41673.stepl.log 

ATTENTION:  0031-408  16  tasks  allocated  by  LoadLeveler,  continuing 
l:Hostname  =f21n03e 
7:Hostname  =f21n04e 
ll:Hostname  =f21n05e 
12:Hostname  =f21n06e 
0:Hostname  =f21n03e 
5:Hostname  =f21n04e 
8:Hostname  =f21n05e 
14:Hostname  =f21n06e 
2:Hostname  =f21n03e 
6:Hostname  =f21n04e 
9:Hostname  =f21n05e 
13:Hostname  =f21n06e 
3:Hostname  =f21n03e 
4:Hostname  =f21n04e 
10:Hostname  =f21n05e 
15:Hostname  =f21n06e 
2:Date  =Thu  Mar  21  09:51:24  CST  2002 
7:Date  =Thu  Mar  21  09:51:24  CST  2002 
8:Date  =Thu  Mar  21  09:51:24  CST  2002 
14:Date  =Thu  Mar  21  09:51:24  CST  2002 
ll:Date  =  Thu  Mar  21  09:51:24  CST  2002 
15:Date  =  Thu  Mar  21  09:51:24  CST  2002 

10:Date  =Thu  Mar  21  09:51:24  CST  2002 
3:Date  =Thu  Mar  21  09:51:24  CST  2002 
5:Date  =Thu  Mar  21  09:51:24  CST  2002 
12:Date  =Thu  Mar  21  09:51:24  CST  2002 
0:Date  =Thu  Mar  21  09:51:24  CST  2002 
6:Date  =Thu  Mar  21  09:51:24  CST  2002 
l:Date  =Thu  Mar  21  09:51:24  CST  2002 
13:Date  =  Thu  Mar  21  09:51:24  CST  2002 

4:Date  =Thu  Mar  21  09:51:24  CST  2002 
9:Date  =Thu  Mar  21  09:51:24  CST  2002 


Job  Step  3  Output 


fl8nl3e%  more  41673.step2.log 
Hostname  =fl5nl3e 

Date  =Thu  Mar  21  09:53:22  CST  2002 

File  Staging  TO  J  ules  Completed  SUCCESSFULLY! 
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VSIPL... continued 


performance  computers.  Efficient 
machine-specific  implementations  of 
BLAS  are  available  for  several  high- 
performance  computers. 

Approach  and  Result 

The  VSIPL  TASP  library  was  modi¬ 
fied  to  call  all  three  libraries  (i.e.r 
FFTW,  LAPACK,  and  BLAS). 

Another  feature  is  a  header  file  that 
allows  the  user  to  switch  between 
building  the  ERI  library  and  the  stan¬ 
dard  TASP  library.  The  final  deliv¬ 
ered  library  will  include  a  perl  script 
so  that  the  end-user  can  download 
each  of  the  reference  libraries  and 
run  the  script  to  add  the  necessary 
modifications.  The  user  will  then 
build  each  library,  and  MSTI  will  pro¬ 
vide  some  example  codes  and 
benchmarking  routines. 

The  initial  results  forVSIPL/ERI  are 
encouraging:  Figure  1  shows  some 
comparative  benchmarks  for  four 
FFT  functions  using  float  and  double 
data  types.  The  top  table  is  the 
VSIPL/ERI  numbers,  and  the  lower 
table  is  the  TASP  numbers.  The  four 
functions  are  real-to-complex  FFT 


out-of-place  (rcfftop),  complex-to-real 
FFT  out-of-place  (crfftop),  complex- 
to-complex  FFT  out-of-place 
(ccfftop),  and  complex-to-complex 
FFT  in-place  (ccfftip).  The  numbers 
were  generated  using  the  Linux 
VSIPL/ERI  implementation  on  an 
800-MHzPIII  with  256M  RAM  (men¬ 
tioned  since  processor  speed  and 
RAM  size  affect  the  performance). 
Figure  2  shows  some  comparative 
Linear  Algebra  benchmarks.  The  six 
functions  are  as  follows:  Cholesky 
decomposition  (symmetric 
(Hermitian)  positive  definite  linear 
system)  for  real  (Cholesky_f)  and 
complex  (Ccholesky_f);  LU  decom¬ 
position  (Gaussian  decomposition) 
for  real  (LU D_f)  and  complex 
(CLUD_f);  general  matrix  product 
(compute  the  general  product  of  two 
matrices  and  accumulate)  Gemp_f; 
and  matrix  product  (product  of  two 
matrices)  M  prod_f.  The  upper  table 
represents  the  VSIPL/ERI  numbers, 
and  the  lower  table  shows  the  TASP 
VSIPL  numbers.  M  ost  of  the  func¬ 
tions  require  two  input  values,  and 
for  these  cases,  the  same  number 
(column  heading  number)  was  used 


for  both  inputs.  These  benchmarks 
were  generated  on  the  same  800- 
M  Hz  Pill  with  256M  RAM. 

Summary  and  Conclusion 

It  is  important  to  note  that  the 
LAPACK  routines  are  called  only 
when  the  matrix  majors  are  column 
and  the  stride  is  equal  to  one,  other¬ 
wise  the  TASP  VSIPL  code  is  execut¬ 
ed.  Itwas  found  that  the  MPROD 
and  GEMP  functions  yielded  no  per¬ 
formance  improvement  over  the 
TASP  VSIPL  implementation.  The 
other  Linear  Algebra  functions 
showed  increased  performance, 
especially  for  the  larger  matrix  sizes. 
Each  of  the  VSIPL/ERI  FFT  functions 
executed  in  half  the  time  of  the  TASP 
VSIPL  FFTs  for  the  sizes  shown.  In 
all,  56  linear  algebra  functions  in 
VSIPL  and  LAPACK/BLAS  were 
wrapped  in  VSIPL/ERI.  VSIPL/ERI 
will  be  free  to  all  U  ,S.  sites 
(Government-purposes  rights)  and 
will  also  be  open  source  for  the  U  ,S. 
Government.  MSTI  will  offer  a  free 
license  for  academic  institutions.  A 
commercial  implementation  will  also 
be  available. 
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